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. Abstract. Wc prove for residually finite groups the following long standing conjecture: 

I the number of twisted conjugacy classes [g ^ hg(j){h^^)) of an automorphism of a finitely 

■ generated group is equal (if it is finite) to the number of finite dimensional irreducible 
^ I unitary representations being invariant for the dual of 0. 

Q_|. Also, we prove that any finitely generated residually finite non-amenable group has the 

I Roo property (any automorphism has infinitely many twisted conjugacy classes). This gives 

■ a lot of new examples and covers many known classes of such groups. 

cn 

1. Introduction 

o, 

The following two interrelated problems are among the principal ones in the theory of 
twisted conjugacy (Reidemeister) classes in infinite discrete groups. The first one is the 
20-years-old conjecture on existence of an appropriate twisted Burnside-Frobenius theory 
(TBFT), i.e. identification of the number -R(0) of Reidemeister classes and the number of 
^ . fixed points of the induced homeomorphism on an appropriate dual object (supposing 
^ \ R{(p) < oo). The second one is the problem to outline the class of i?oo groups (that is 

I = oo for any 0). 

^ ■ In this paper important advances in both problems are obtained. Namely, first, it is proved 

. that TBFT holds for finitely generated residually finite groups (we take the finite-dimensional 
Tj- I part of the unitary dual as the dual space). Secondly, it is discovered that finitely gener- 
O ' ated residually finite non-amenable groups are -Roo-groups. Several supplementary results of 
independent interest are obtained. 

The interest in twisted conjugacy relations has its origins, in particular, in the Nielsen- 
Reidemeister fixed point theory (see, e.g. [66, 27]), in Selberg theory (see, eg. [101, 1]), and 
Algebraic Geometry (see, e.g. [57]). In representation theory twisted conjugacy probably 

■ occurs first in Gantmacher's paper [47] (see, e.g [102, 88]). 
The problem of determining which classes of groups have the -Roo property is an area 

of active research initiated in [29]. In some situations the Roo property (for fundamental 
groups) has direct topological consequences. For example, using this property in [54] for any 
n ^ 4 a compact nilmanifold M, dim M = n, is constructed, such that any homeomorphism 
/ : M — M is homotopic (for n ^ 5 isotopic) to a fixed point free map. In [20] it is 
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shown that an infra-nilmanifold that admits an Anosov diffeomorphism cannot have the i?oo 
property since the Reidemeister number of an Anosov diffeomorphism on an infra-nilmanifold 
is always finite. Also, as we will see below it can be used to detect the trivial knot. 

The Reidemeister class of 5^ e G we will denote by {g}(j,- {g}^ '■= {xg(p{x~^) G G | x G G}. 
A (f)- class function is any function on G, which is constant on Reidemeister classes, or, 
equivalently, which is invariant under the twisted action g H- xg(f){x~^) of G on itself. 

Definition 1.1. Denote by G the set of equivalence classes of unitary irreducible represen- 
tations of G and by G/ its part corresponding to finite-dimensional representations. The 
class of p in G we will denote by [p]. An automorphism </> of G induces a bijection : G — ?■ G 
by the formula [<t>{p)] '■= [p o (f)]. 

The following (first) main theorem is proved as Theorem 4.1 below. 
Theorem A. Let (p : G ^ G be an automorphism of a residually finite finitely generated 
group G with R{4>) < 00. Then R{(j)) is equal to the number of cf)- fixed points on Gf. 

This theorem covers the known case of almost polycyclic groups [33] and contains f.g. 
metabelian groups and restricted wreath products. Some known examples of with R{(f)) < 
00 are listed in Section 8. 

In the case of |G| < 00 and = Id it becomes the celebrated Burnside-Frobenius theorem, 
which says that the number of conjugacy classes is equal to the number of equivalence classes 
of irreducible unitary representations of G. In [29] it was discovered that this statement 
remains true for any automorphism of any finite group G. Indeed, R{(f)) is equal to the 
dimension of the space of 0-class functions on this group. Hence, by Peter- Weyl theorem, 
R{(f)) is identified with the sum of dimensions dp of twisted invariant elements of End(ifp), 
where p runs over G, and the space of a representation p is denoted by Hp. By the Schur 
lemma, dp = 1, if p is a fixed point of 0, and is zero otherwise. Hence, R{(f)) coincides with 
the number of fixed points of 0. 

Theorem A has important dynamical consequences (see [27, 32] for an extended discus- 
sion). In particular, suppose, i?(0") < 00, ^ 1, in Theorem A. Then we have for all n 
the following Gauss congruences for the Reidemeister numbers, which are important for the 
theory of the Reidemeister zeta function: 

(1) /^(^) ■ ^(0"^^^) = mod n, 

d\n 

where fi{d), G N, is the Mobius function. Indeed, apply the Mobius' inversion formula to 
the following evident consequence of Theorem A: R{(f)"-) = # Fix(0"|g^) = X]d|n -^rf' where 

Pn denote the number of periodic points of on G/ of least period n. We obtain (1) with 
Pn on the right. But Pn is always divisible by n, because P„ is exactly n times the number 
of orbits of length n. 

Formula (1) was previously known in the special case of almost polycyclic groups (see 
[32, 33], where more detail can be found). The history of the Gauss congruences for integers 
can be found in [111]. Gauss congruences for Lefschetz numbers of iterations of a continuous 
map were proved by A. Dold [23]. 

The second main theorem (proved as Corollary 5.6 below) is as follows. 
Theorem B. Any non-amenable residually finite finitely generated group is an R^o group. 
(It is enforced in Prop. 5.8). 
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This gives a lot of new examples of groups with -Roo-property. (A list of known cases of 
Roo and non-i?oo groups can be find in Section 8, to not overload the Introduction.) 
Among the new classes we would like to emphasize the following important ones. 

(1) Irreducible infinite non-affine finetely generated Coxeter groups. Indeed, they are 
linear [106], hence residually finite [75]. They contain a subgroup which maps onto 
a non-amenable group [76], thus they are non-amenable. 

(2) Lattices (finitely generated cocompact discrete subgroups) G in direct products Gi x 
■ ■ ■ X Gm of linear groups over fields fci, of zero characteristic, where at least one 
of Gi, say Gi, is a non-compact semi-simple Lie group. Indeed, Gi is non-amenable 
by [44] (see [6, p. 432]), hence Gi x ■ ■ ■ x Gm is non-amenable, and G is non-amenable 
(see, e.g. [6, Corollary G.3.8]). By the same argument as in the previous item, G is 
residually finite. This class contains ^-arithmetic lattices, in particular groups from 
(7), Sect. 8. 

(3) Fundamental groups of compact 3-dimensional manifolds and knot groups with few 
exceptions. They are finitely generated and residually finite by [61] and Perelman's 
proof of Thurston's geometrization conjecture [104]. More precisely, the JSJ de- 
composition (see, e.g. [60, 98]) gives parts, which are either Seifert manifolds, or 
hyperbolic. The fundamental group is a graph product of fundamental groups of 
these parts. Thus, by Theorem B, the group under consideration is Roo group if it is 
non-amenable. Such groups are characterized geometrically (via bounded cohomol- 
ogy [64]) in [43, 42]. 

In particular, via existence/non-existence of Roo property for knot groups we can 
detect unknot. Indeed, since the trivial knot's group is abelian, it is not -Roo, while 
the other ones have infinite-dimensional second bounded cohomology [43, 42]. Thus, 
they are non- amenable and Roo groups by Theorem B. 

(4) Several important classes are among non-linear groups: 

(a) Periodic Golod-Shafarevich groups are finitely generated residually finite non- 
amenable (having quotients with property (T)) [24] (see also [25]). A linear 
non-amenable group contains F2 (Tits alternative, [109, pp. 145-146]). Thus, 
periodic Golod-Shafarevich groups are non-linear. 

(b) Similarly, finitely generated infinite torsion groups with positive rank gradient 
from [89] or positive power p-deficiency from [96] (see also [25, Sect. 9]) are -Roo 
groups. 

(c) Aut(F„), 77, ^ 3, is finitely generated [86], residually finite (Baumslag, see [5, Sect. 
2]), non-amenable and non-linear [40]. The same is true for Out (-Fn), n ^ 4 (see 
e.g. [107]). 

Also, the theorem covers iterated semidirect products -F^, x ■ ■ ■ x Fd^ of free groups and its 
finite extensions (poly-fg-free- by-finite groups), because they are residually finite (see, e.g. 
[80, Theorem 7, p. 29]) and evidently non-amenable (having a non-amenable subgroup) and 
finitely generated. This class contains Artin's pure braid groups, the fundamental group of 
the complement of any affine fiber-type hyperplane arrangement and some others, see e.g. 
[14, 79]. 

On the other hand, the new result covers (up to the conjecture about the residually 
finiteness) the most extended known class, non-elementary Gromov hyperbolic groups (1) in 
Sect. 8. 

Some other new classes are discussed in Section 8. 
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In Section 3 we establish important inequalities, relating the number of fixed points of 
and R{4>). We prove Theorem A in Section 4 and Theorem B in Section 5. In Section 6 we 
introduce (ir)rational representations and describe their connection with twisted conjugacy 
classes. In Section 7 we introduce and investigate a new class of groups: 5*00 groups. 

Acknowledgments. The present paper is a part of our joint research programm in Max- 
Planck-Institut fiir Mathematik (MPIM) in Bonn. We would like to thank the MPIM for 
its kind support and hospitality while the greater part of this work was completed. The 
first author is indebted to the Institute for Advanced Study, Princeton, and Institut des 
Hautes Etudes Scientifiques (Bures-sur-Yvette) for for their support and hospitality and the 
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The authors are grateful to V. Manuilov, A. Shtern, and A. Vershik for important dis- 
cussions on non-commutative harmonic analysis aspects of the present research and to M. 
Abert, V. Bardakov, P. Bellingeri, D. Gongalves, R. Grigorchuk, M. Kapovich, V. Nekrashe- 
vich, D. Osin, G. Prasad, and M. Sapir for helpful comments and valuable suggestions on 
different group-theoretic questions. The paper was partially inspired by the excellent survey 
[19] by P. de la Harpe and the interesting paper [65] by E. Jabara. 

2. Preliminary Considerations 

We start from the following estimation (supposed to be known to E. Landau (cf. [71]) and 
R. Brauer (cf. [12])) (cf. [84]). 

Lemma 2.1. Let Xi ^ X2 ^ . . . ^ x„ be positive integers such that X]r=i(-^«)^^ ~ Then 
Proof. One has 

(2) xi ^ n, 

since otherwise Yl^=i < Xl^Li n ~ '^^^ write with some positive integers i/r for 

1 ^ r ^ n — 1: 



n — r \ - 1 X ^ 1 



i=r+l 1=1 

(3) nxiX2 . . . Xr ^ {n — r)xiX2 . . . ^ a^r+i- 

By induction, (2) and (3) imply the statement. □ 

With some additional efforts the estimation can be improved (cf. e.g. [84]) but we do not 
need this. Let us note that, since logg?^ < n, one has 

(4) n"-^ < n" = 2"'°S2 « < 2"'. 

Denote the stabilizers related to the twisted action of G on itself by 

Stl-{g,h) ■.= {keT I kg<j>{k-') = h}, Stl^ig) := Stl^ig^g). 

In particular, 

St'^^ie) = {ker\ k<p{k~') = e} = Cg(0) 
(fixed elements of 0). Evidently, St^^{g, h) = ii h ^ {9}<f>- Otherwise 

St'J^ig, sg(P{s~')) = {keT\ kg^{k-') = sg(P{s"')} = {keT\ s~'kg^{k-h) = g}, 
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i.e. St^^{g, sg(f){s~^)) = s ■ St^^{g) is a coset of this group. Thus 

(5) \St'^{g,sg<P{s~^))\ = \St'-{g)\. 

An evident consequence of the above statement is the following lemma (see [65, Lemma 
4])- 

Lemma 2.2. Let \G\ < oo, r = R{(j)), (p-.G^G. Then \Gg{^)\ ^ r^"^ 
Proof. Let {gi}(f, be distinct Reidemeister classes, i = 1, . . . ,r, gi = e. Then 

\G\ 



Dividing by IGj and applying Lemma 2.1 we obtain 

ICcm = \Stl^{e)\ ^ m^x\Stl^{g,)\ ^ r^-\ 

□ 

The following fact is well known. 

Lemma 2.3. Let G be finitely generated, and H' G G its subgroup of finite index. Then 
there is a characteristic subgroup H d G of finite index, H G H' . 

Proof. Since G is finitely generated, there is only finitely many subgroups of the same index 
as H' (see [59], [70, § 38]). Let H be their intersection. Then H is characteristic, in particular 
normal, and of finite index. □ 

Let us denote by : G — G the automorphism Tg{g) = gg g^^ for g E G. Its restriction 
to a normal subgroup we will denote by Tg as well. 

Lemma 2.4. {g}^k = {gk}r^_^o^. 

Proof. Let g' = f g <f){f~^) be 0-conjugate to g. Then 

g'k = fg 0(r 1) k = fgkk-' <f>{f~') k = f{gk) o 0) (r i). 

Conversely, if g' is (r^-i o 0)-conjugate to g, then 

g'k^' = f g {r,-. o ct^){f~^)k~^ = fgk~'<j>{f~'). 

Hence a shift maps </)-conjugacy classes onto classes related to another automorphism. □ 

Corollary 2.5. i?(0) = R{Tg o 0). 

If C G is a 0-invariant and normal, the quotient map induces an epimorphism of 
Reidemeister classes, in particular, 

(6) i?G(</') > Rg/h($) 

for the induced automorphism (p '■ G/H — G/H (see [48, 49, 50] and [33, Sect. 3] for details 
and refinements). 

Definition 2.6. A group G is called residually finite if for any finite set K G G there exists 
a normal group H of finite index such that H D K = 0. Taking K formed by g^^gj for 
some finite set Kq = {gi, . . . , gs} one obtains an epimorphism G — ?■ G/H onto a finite group, 
which is injective on Kq. 
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Remark 2.7. If a residually finite group G is finitely generated, then groups H in the 
definition can be supposed to be characteristic (by Lemma 2.3). 

We will remind now some facts from Representation Theory and Harmonic Analysis (see 
[22] and [6] for an effective introduction). (Left) regular representation Ac is the unitary 
representation of G on £'^{G) by left translations. The completion C^(G) of i^{G) by the 
norm of B{i'^{G) is called reduced group C* -algebra of G. The completion G*{G) of i^{G) by 
the norm of all unitary representations is called (full) group C* -algebra of G. The algebra 
Gl{G) is a quotient of G*{G). 

Non-degenerate representations of G*{G) are exactly unitary representations of G, in 

particular, G*{G) = G. For a representation p of G we denote by G*p the corresponding 
representation of G*{G), and by C*Kerp the kernel of G*p. One introduces on G the 
Jacobson-Fell or hull-kernel topology defining the closure of a set X by the following formula 

X = {[p] : C*KerpD |J C*Ker7r}. 

This topology can be described in terms of weak containment: a representation p is weakly 
contained in representation vr (we write p -< tt) if diagonal matrix coefficients of p can be 
approximated by linear combinations of diagonal matrix coefficients of vr uniformly on finite 
sets. Here a matrix coefficient of a representation p on a Hilbert space H is the function 
g I— >■ {p{g)^, rj) on G for some fixed ^,7] ^ H, and a diagonal one corresponds to ^ = 77. Then 
G* Ker n (Z G* Ker p if and only if p -< tt. Since 

G* Ker(pi © ■ ■ ■ © p„) = n^.G* Ker p,, 

(7) Pl © ■ ■ ■ © Pm -< TT if Pi -< TT, Z = 1, . . . , m. 

An amenable group may be characterized in several equivalent ways (see e.g. [6]), in 
particular: 

• There exists an invariant mean on i°°{G). 

• 1q -< Ac, where 1^ is the trivial 1-dimensional representation. 

• c*{G) = Gl{G). 

A group G has property (T) if the trivial 1-dimensional representation Iq is an isolated 
point of G and is G*-simple if p -< A implies A -< p for any p [6, 19]. 

Definition 2.8. A function / : G — t- C is called positively definite function if for any finite 
collection {gi, . . . ,gs} C G the matrix ||/(fi'j"^5'j)|| is non-negatively definite. 

Definition 2.9. Fourier- Stieltijes algebra B{G) can be defined in three equivalent ways: 

(1) B{G) is formed by finite linear combinations of positively definite functions; 

(2) B{G) is formed by all matrix coefficients of all unitary representations of G; 

(3) bIg) = G*{Gy, i.e. it is the Banach space dual to G*{G). 

Remark 2.10. Matrix coefficients of distinct finite-dimensional representations are linear 
independent (see [16, Corollary (27.13)] for an algebraic argument). Another argument uses 
the fact that these representations are disjoint having maximal ideals as kernels. Passing to 
the weak containment interpretation gives the linear independence. 

Let us remind now some facts from [32, 33]. 
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Lemma 2.11. Let p be a finite dimensional irreducible representation of G on Vp, and 
(f) : G ^ G is an automorphism. 

1) . There exists a twisted invariant function : G — t- C being a matrix coefficient of p if 
and only if (f)[p] = [p] . 

2) . In this case such uj is unique up to scaling. 

3) . The space Kp := {h G EndVp \ h = a — p{g)ap{(j){g~^) for some g & G and some 
a G EndVp} has codimension 1 if (t)[p] = [p] and coincides with End otherwise. 

4) . If we have several distinct (p-fixed classes, then the correspondent twisted invariant 
functions are independent. 

Proof. Let us sketch a proof, the details can be found in [32, 33, 105]. Matrix coefficients of 
a finite dimensional representation p arise from functionals on End Vp and can be written as 
g H- Trace(ap((y')) for some matrix a G EndV^. Since the equality 

= Trace(a6) — TTa,ce{ap{h)bp{(f){h~^))) = Trace((a — p{(f){h'~^))ap{h))b) 

for any b and h implies p{(f){h))a = ap{h), the above matrix coefficient is twisted invariant 
if and only if a is an intertwining operator between p and <f) o p- This gives 1), and Schur's 
lemma gives 2). Now, 3) follows immediately from 1) and 2), because Kp is evidently the 
intersection of kernels of all twisted invariant functionals on End Vp. Finally, 4) follows from 
Remark 2.10. □ 

3. Fixed points and Reidemeister numbers 

Theorem 3.1 (an extraction from [65]). Let T be a finitely generated residually finite group. 
If Gr{<f)) = Fix(0) is infinite, then R{(f)) = oo. 

Proof. By Lemma 2.2 and inequality (4) for any automorphism if) of a finite group G 
(8) ^log2 ICgWI ^i?GW. 

Let {xi,X2, . . . } = Cr(0). Then for every n we can find (see Remark 2.7) a characteris- 
tic subgroup r„ of finite index in F such that the quotient map p„ : F — j- F/F„ =: G„ 
is injective on Let 0„ : G„ — ?■ G„ be the induced automorphism. Then 

{pn{xi), . . .,Pn{xn)} C Cg,^(0„), hcuce (6) and (8) imply 



i?r(0) > RgMu) ^ Vlog2|CG„(0n)| > Vlog2n. 
Since n was arbitrary, we are done. □ 

Corollary 3.2. Suppose, T is a torsion-free finitely generated residually finite group. If 
Cr(0) > 1, then R{(j)) = oo. 

Proof. Indeed, in this case F has an infinite subgroup= Z formed by fixed points. □ 

Theorem 3.3. Let F be as in Theorem 3.1 and R{(f)) < oo. Then \St^^{g, sg(f){s~^))\ < oo 
for any g,s & 

Proof. By (5) we need to estimate \St*^{g)\ only. Under the left translation by g~^ the 
Reidemeister class {g}(p will be moved to the class {e}^ __^o<f> (see Lemma 2.4). Thus 

\St',-i9)\ = \Stl\.,{e)\ = \Crir,-,o^)\. 



a 

If it is infinite, by Theorem 3.1 and Corollary 2.5 we have oo = R{Tg-i o 0) = R{(j)). A 
contradiction. □ 
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Corollary 3.4. Let T and be as in Theorem 3.3 and \T\ = oo. Then all Reidemeister 
classes of (p are infinite. 

Proof. Indeed, |r| = \{g}4 ■ \Stl^{g)\. □ 

4. Twisted Burnside-Frobenius theorem 

Theorem 4.1. Let R{(j)) < oo for an automorphism (p of a finitely generated residually finite 
group G. Then R{4>) is equal to the number of finite dimensional fixed points of (p. 

Proof. R{(j)) equals the dimension of the space of twisted invariant elements of i°°{G), i.e. 
functionals on i^{G) such that their kernels contain the closure Ki in i^{G) of the space of 
elements of the form b — g[b], g[b]{x) := b{gx(j){g~^) . 

Since R{4>) < oo, codim A'l = R{4>), and Ki has a Banach space complement of dimension 
R{4>)- We can take it in a way such that it has a base G C[G], i = 1, . . . , R{(f)), i.e., all 
Oj's have a finite support. Let p : G F = G/H he an epimorphism on a finite group 
F such that it distinguishes all elements from the union of (finite) supports of a,j and H is 
characteristic (see Remark 2.7). The image of £^{G) under the induced homomorphism pi 
is i^{F) = C[F]. Also Ki maps epimorphically onto the space Kp of elements (3 — p{g)[(3] = 
Viip) — p{g)[pi{b)] = piip — g\b]) in C[F]. Thus, {pi(ai)} form a basis of a complement to 
Kp in C[F]. 

Decompose this (finite dimensional) algebra C[-F] into a direct sum of matrix algebras, 
i.e., decompose the image pi{l^{G)) (in fact, pi{C* {G))) into blocks: 

J ■.C[F] = ®ti^ndVi, 

where (i = 1, . . . , A^) are some irreducible representations on Vi. Since this decomposition 
is in fact associated with the decomposition of the left regular representation of 

these Pi's are pairwise non-equivalent [99, 1, §2]. Let Ki be formed by x — pi{g)[x\ in EndV^. 
Since J is an algebra isomorphism, 

i?(0) = codimi^i = ^^codimi^j. 

i 

The last one is 1 if </'(pi) = Pi (in this case intertwining operators give a linear complement 
to Ki) and otherwise by Lemma 2.11. Thus, -R(0) ^ the number of finite dimensional fixed 
points of (j). 

The inverse estimation follows from Lemma 2.11. □ 

Remark 4.2. We use here an approach from [105]) with 1^{G) instead of G*{G). 

Remark 4.3. As it is known (see e.g. [94, 110]) f.g. residually finite groups (e is closed 
in the profinite topology) form a strictly larger class then conjugacy separable groups (each 
conjugacy class is closed). Our argument is not applicable to a "proof" of a wrong fact 
(coincidence of these classes). Indeed, we have no map to identify invariant elements from 
1°°{G) with a complement to Ki. So, our argument uses in a crucial way the finiteness 
of R{4>), while the number of usual conjugacy classes is typically infinite. We have an 
infinite number of independent characters of finite dimensional irreducible representations 
for a residually finite group and can separate infinitely many ordinary conjugacy classes, but 
not necessary all of them. 
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Corollary 4.4. In the situation of Theorem 4-i (p-class functions are in B{G), Fourier- 
Stieltijes algebra of G {c.f. the discussion in [105]). 

A group G is 0-conjugacy separable, if there is a homomorphism onto finite group G/H 
with a 0-invariant normal subgroup H inducing a bijection of twisted conjugacy classes. 

Corollary 4.5. In the situation of Theorem ^.1 G is (p-conjugacy separable. 

One can slightly modify the proof of Theorem 4.1 to obtain the following statement. 

Proposition 4.6. Suppose, R{(j)) = oo for an automorphism (f) of a finitely generated resid- 
ually finite group G. Then the number of finite dimensional fixed points of (j) is infinite. 

Proof. Let M be an arbitrary positive integer. Consider a subspace L C 1^{G) such that 
Ki G L and codim(L) = m. Choose a complement to L with a base {ai, . . . , am} of finitely 
supported functions and corresponding F and H as in the proof of Theorem 4.1. In the 
same way Ki maps onto Kp and the images of form a part of a basis of a complement to 
Kp in C[F]. We obtain m ^ the number of finite dimensional fixed points of (p. Since m is 
arbitrary, we obtain the statement. □ 

5. Amenability, twisted inner representation, and _Roo property 

Our argument here partially follows [68]. 

Definition 5.1. Denote by 7^ the twisted inner representation of G on i^{G), i.e. 

7t{x){f){g) = f{xg<p{x-')), x,geG, f e f{G). 

Denote G^{a) := St^^{a), a E G. Evidently, 7^ decomposes into a direct sum of represen- 
tations 7^ being restrictions of 7^ onto {a}^ (i.e. on ["^{{a}^)). 

Lemma 5.2. The representation 7^ is equivalent to the induced representation ind^^^^-, lc0(a) ■ 

Proof. Indeed, this induced representation T can be realized on £^(C0(a)\G) by the following 
action \r{g){f)]{x) = f{xg), x G C(^(a) \G, g E G, where C(^(a) \G is the space of left cosets 
by G^{a). Let us identify C^{a) \ G with {a}^ by i{G^{a) ■ g) = ^Q{g){a). Evidently, this 
map is well defined and gives a unitary isomorphism 

/ : e{{a}^) ^ e{G^{a) \ G), J(/)(x) := f{i{x)). 

Then 

[lo^tmmx) = bt{g){f)Mx)) = f{ghacf>{{gh)-')), x = C^{a) ■ h, 
[Tig) o J(/)](x) = Iif)ixg) = f{i{xg)) = f{it{hg){a)) = f {gha(t>{{gh)-^)) . 
Thus, I is an intertwining unitary. □ 

Theorem 5.3. Suppose, \G^{a)\ < 00, for any a E G. Then 7^ is weakly contained in the 
regular representation Xq. 

Proof. The characteristic functions Xc^(a), a E G, are positively definite functions associated 
to Ag, because they are finite sums of translations of 6e- Hence, ind^^(^-) Ic^ia) -< (cf. [6, 
E.4.4]). By Lemma 5.2 and the decomposition of 7^ we obtain 7^, -< Ag'. □ 

Theorem 5.4. Suppose, G is a finitely generated residually finite group and R{4>) < 00 for 
some (f) : G ^ G. Then G is amenable. 
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Proof. In this case (see Theorem 3.3 above) |C(^(a)| < oo, for any a & G. Thus, by Theorem 
5.3 7^ -< Xg- So, it is sufficient to verify that 1^ -< '~fQ, i.e. C* Ker 7^ C C* Ker 1^. 
Suppose, / = Y^gacf^^g ^ C*Kei-if., i.e. 

(9) f'^tWh = Yl f'LgR^{9~^)5h = Y f%hH9--) = for any h e G. 

g£G geG g&G 

Since gh(f){g~^) = gih(f){g^^) if and only if gi,g2 G St^^{h,x) for some x G {h}^, (9) is 
equivalent to 

(10) Yl /' = 0' 

any h e G and x G {h}^. 

g£Stfih,x) 

Take any h, e.g. h = e. Then St^^{e,e) = G^{e) is a finite subgroup of G and the 
other St^^{h,x) form the set of all cosets with the respect to this subgroup. This implies 
J2geG — ^ (fo^ reasonable convergence, because the subgroup is finite). In particular, 
/ G C* Ker Iq and Iq ^ it ^ ^g- □ 
Remark 5.5. Main results of [65] illustrate this theorem. 

Corollary 5.6. Any non-amenable residually finite finitely generated group is an Roo group. 

Remark 5.7. In fact we have used only the estimation |C(^(a)| < 00, for any a E G. This can 
occur not only when the group is residually finite. The following case is especially important 
for applications. 

Suppose, we have an exact sequence 

(11) ^ F ^ G ^ H = G/F ^0, \F\=m< 00, 

G is finitely generated, H is residually finite and non-amenable, F not necessary character- 
istic, in particular, 0-invariant (otherwise we have the Roo property with the help of epimor- 
phity of mapping of Reidemeister classes). As in Theorem 3.1, suppose {xi,X2, • • • } = Gg{(P) 
is infinite. Let if„ be a characteristic subgroup of finite index in H such that the quotient 
map pn : H ^ H/Hn =■ F^ is injective on {p{xi), . . . ,p{xn)}- Then Gn '■= Ker(p„ op) is a 
subgroup of G of index As above, we can assume Gn to be characteristic, since G is 

finitely generated. Denote s := #{p(xi), . . . ,p{xn)}- Thus, s ^ n/m. Then 



i?G(0) ^ RfMti) > A/log2 |Cir„(0„)| ^ Vlog2 s ^ v^loggn - logam. 

Since n was arbitrary, we have Rg{4>) = 00. Thus, if Rg{(P) < 00, then 6*0(0) < 00. Hence, 
G^{a) < 00, and by Remark 5.7 we obtain the following statement. 

Proposition 5.8. Suppose, we have {11), where G is finitely generated, H is residually finite 
and non- amenable, \F\ < 00. Then G is an Roo-group. 

Another large class of examples arises from 

Proposition 5.9. Let G be a normal finitely generated subgroup of a residually finite G*- 
simple group. Then G is an Roo-group. 

Proof. Since G is non-amenable (see e.g. [19, Prop. 3]), G enter conditions of Corollary 
5.6. □ 
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6. Rational points 

In this section we show that not every finite- dimensional representation can be fixed by cf) 
if R{<j)) < oo. 

Definition 6.1. Let [p] G G/, i— t- Tg be a (class of a) finite-dimensional representation. 
We say that p is rational if the number of distinct T^'s is finite, and irrational otherwise. 

Remark 6.2. Evidently, p is rational if and only if it can be factorized through a homo- 
morphism G — )■ F on a finite group. 

As a consequence of the proof of Theorem 4.1 (see also Corollary 4.5) we obtain the 
following statement. 

Theorem 6.3. Suppose, G is a finitely generated residually finite group, and is its auto- 
morphism with R{(j)) < oo. Then no irrational representation is fixed by 0. 

Theorem 6.4. Let G be a finitely generated group and for an automorphism at least one 
of the following two conditions holds: 

1) . There exist infinitely many finite- dimensional representation classes in G fixed by 0. 

2) . There exists an irrational representation p fixed by 0. 
Then R{(t>) = oo. 

In particular, if we have one of these conditions for every automorphism 0, then G has 
Roo property. 

Proof. 1) This follows immediately from Lemma 2.11. 

2) Suppose that fp{g) = TTa.ce{ap{g)) (see Lemma 2.11) takes finitely many values and 
will arrive to a contradiction. Indeed, fp is a non-trivial matrix coefficient. Hence, (see, e.g. 
[69]) its left translations generate a finite-dimensional representation, which is equivalent 
to a direct sum of several copies of p. The space W of this representation has a basis 
Lg-^fp,...,Lgi^fp. Thus, all functions from W take only finitely many values (with level 
sets of the form (ligiUj, where Uj are the level sets of fp). Taking unions of these sets (if 
necessary) we can form a finite partition G = ViU ■ ■ ■ U such that elements of W are 
constant on the elements of the partition and for each pair Vi ^ Vj there exists a function 
from W taking distinct values on them. Thus any left translation maps Vi onto each other 
and the representation G on W factorizes through (a subgroup of) the permutation group 
on m elements, i.e. a finite group. The same is true for its subrepresentation p, thus it is 
rational. A contradiction.^ □ 

7. ISOGREDIENCE CLASSES 

Definition 7.1. (see [73]) Suppose, $ G Out(G) := Aut(G)/ Inn(G). We say that a, /3 G $ 
are isogredient (or similar) if (3 = Th o a o for some h E G. 

Let S{^) be the set of isogredience classes of $. If $ = Idc, then above a and /3 are inner, 
say a = Tg, f3 = Ts. Since elements of center Z{G) give trivial inner automorphisms, we may 
suppose g,s E G/Z{G). Then the equivalence relation takes the form = Thgh-^, i-e., s and 
g are conjugate in G/Z{G). Thus, iS(Id) is the set of conjugacy classes of G/Z{G). 

Denote by S'($) the cardinality of iS($). 



Probably this argument is known, but wc have not found an appropriate reference. 



12 



ALEXANDER FEL'SHTYN AND EVGENIJ TROITSKY 



For a topological motivation of the above definition of the isogredience suppose that G = 
7ri(X), X is a compact space, and $ is induced by a continuous map f : X ^ X. Let 
p : X ^ X he the universal covering oi X , f : X ^ X a lifting of /, i.e. p o f = f o P- 
Two liftings / and /' are called isogredient or conjugate if there is a 7 G G such that 
/' = 70/0 7~^. Different lifting may have very different properties. Nielsen observed 
[85] that conjugate lifting of homeomorphism of surface have similar dynamical properties. 
This led him to the definition of the isogredience of liftings in this case. Later Reidemeister 
[93] and Wecken [108] succeeded in generalizing the theory to continuous maps of compact 
polyhedra. 

The set of isogredience classes of automorphisms representing a given outer automorphism 
and the notion of index Ind($) defined via the set of isogredience classes are strongly related 
to important structural properties of $ (see [46]), for example in another (with respect to 
Bestvina-Handel [8]) proof of the Scott conjecture [45]. 

One of the main results of [73] is that for any non-elementary hyperbolic group and any 
$ the set i5($) is infinite, i.e., «S'($) = 00. We will extend this result. First, we introduce an 
appropriate definition. 

Definition 7.2. A group G is an Soo-group if for any $ the set iS($) is infinite, i.e., 5'($) = 
00. 

Thus, the above result from [73] says: any non-elementary hyperbolic group is an Sca~ 
group. On the other hand, finite and Abelian groups are evidently non Soo-groups. 

Now, let us generalize the above calculation for $ = Id to a general $ (see [73, p. 512]). 
Two representatives of $ have form o a, o a, with some s, g G G. They are isogredient 
if and only if 

T^O a = TgOTsO aOTg^ = TgOTsO r^(g-l) o a, 

Tq = 9 = 9sa{g~^)c, c G Z{G). 

So, the following statement is proved. 

Lemma 7.3. 5'($) = Rg/z(G){p) , where a is any representative of ^ and a is induced by a 
on G/Z{G). 

Since Z{G) is a characteristic subgroup, we obtain from Lemma 7.3 and epimorphity (see 
before (6)) the following statement (in one direction it was discussed in [50, Remark 2.1]). 

Theorem 7.4. Suppose, \Z{G)\ < oo. Then G is an Roo-group if and only if G is an 
Soo- group. 

Remark 7.5. Of course, this argument is applicable to an individual $ as well. 

Theorem 7.4 and Corollary 5.6 imply 

Corollary 7.6. Suppose, G is a finitely generated non-amenable residually finite group with 
\Z{G)\ < oo. Then G is an S^o-group. 

Corollary 7.7. Suppose, G is a finitely generated G* -simple residually finite group. Then 
G is an S^o- group. 

Proof. In this case the center is trivial and the group is non-amenable (see e.g. [19]). □ 

Now we can give a more advanced example of a non S'oo-group. Namely, consider Osin's 
group [90]. This is a non-residually finite exponential growth group with two conjugacy 
classes. Since it is simple, it is not 6*00 by Theorem 7.4. 
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8. Examples 

8.1. Known examples of Roo groups. It was shown by various authors that the following 

groups have the -Roo-property: 

(1) non-elementary Gromov hyperbolic groups [28, 73]; relatively hyperbolic groups [35]; 

(2) Baumslag-Solitar groups BS{m,n) except for BS{1,1) [36], generalized Baumslag- 
Solitar groups, that is, finitely generated groups which act on a tree with all edge 
and vertex stabilizers infinite cyclic [72]; the solvable generalization F of BS{l,n) 
given by the short exact sequence 1 Z[^] — F — Z'^ — i- 1, as well as any group 
quasi-isometric to F [103]; 

(3) a wide class of saturated weakly branch groups (including the Grigorchuk group 
[55] and the Gupta-Sidki group [58]) [31], Thompson's group F [10]; generalized 
Thompson's groups Fn^o and their finite direct products [52]; 

(4) symplectic groups Sp{2n, Z), the mapping class groups Mods of a compact oriented 
surface S with genus g and p boundary components, 3g + p — 4 > 0, and the full 
braid groups Bn{S) on n > 3 strings of a compact surface S in the cases where S 
is either the compact disk D, or the sphere [37], some classes of Artin groups of 
infinite type [67]; 

(5) extensions of SL{n,Z), PSL{n,Z), GL{n,Z), PGL{n,Z), Sp{2n,Z), PSp{2n,Z), 
n > 1, by a countable abelian group, and normal subgroup of SL{n,Z), n > 2, not 
contained in the centre [81]; 

(6) GL{n,K) and SL{n,K) if n > 2 and K is an infinite integral domain with trivial 
group of automorphisms, or K is an integral domain, which has a zero characteristic 
and for which Ant{K) is torsion [83]; 

(7) irreducible lattice in a connected semi simple Lie group G with finite center and real 
rank at least 2 [82]; 

(8) some metabelian groups of the form x Z and Z[l/j9]" xi Z [38]; lamplighter groups 
ZnlZ a and only if 2|n or 3\n [53]; free nilpotent group N^c of rank r = 2 and class 
c ^ 9 [54], Nrc of rank r = 2 or r = 3 and class c > 4r, or rank r > 4 and class 
c > 2r, any group for c > 4, every free solvable group 5*2* of rank 2 and class 
t > 2 (in particular the free metabelian group M2 = 6*22 of rank 2), any free solvable 
group Srt of rank r > 2 and class t big enough [95]; some crystallographic groups 
[21, 74]. 

8.2. Coverage by Theorem B. Coverage of items (1) and (7) is discussed in Introduction. 
Non-elementary (residually finite) relatively hyperbolic without finite normal subgroups (in 
particular, hyperbolic without torsion) groups are C*-simple [2, 18]. Thus their f.g. normal 
subgroups are -Rqo groups by Proposition 5.9. 

This observation can be extended to the following large class introduced in [17]. Suppose 
that a residually finite finitely generated group G contains a non-degenerate hyperbolically 
embedded subgroup [17]. Then it is non-amenable [17, Theorem 8.5(b)], hence, -Roo by 
Theorem B.^ If G enters conditions of [17, Therem 8.11], for example, if G has no nontrivial 
finite normal subgroups, the Roo property holds for any its finitely generated residually finite 
subgroup by Proposition 5.9. 

Items (2), (3), (8) are not covered. 

^D.Osin has communicated to us that he has another proof that groups with non-degenerate hypcrbohcaUy 
embedded subgroup are Roo groups. 
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Most part of groups from item (5) is covered by Theorem B. Linear groups are residually 
finite (using Z — )■ Z//cZ on elements). Taking the quotient by the center commutes with this 
map. Thus, projective hnear groups are residually finite as well. They are finitely generated 
and non-amenable. Indeed, all of them have the free group F2 as a subgroup (for linear 
groups this is a direct fact and the center evidently does not meet this -F2, hence for the 
projective as well). 

Groups from (4) are covered by Theorem B. For the symplectic group it is explained 
above. Braid groups = Bn{D'^) are residually finite finitely generated. Indeed, one can 
use either Artin's representation in Aut(F„) (see [92, 2.19-2.20]), or to use the exact sequence 
En ^ Ni^n ^ ■ ■ ■ ^ Nn,n = 6 with all quoticuts being free groups except of the first one 
which is the (finite) symmetric group (see, e.g. [77]). This sequence implies non- amenability 
of Bn for n ^ 3. 

Mapping class groups for compact oriented surfaces are residually finite [56] and are non- 
amenable for 3g + p — 4 > 0. Indeed, these groups act on hyperbolic space [78, 11] in such a 
way that [7] is applicable. Some other cases can be added after an individual analysis. The 
case of Bn{S'^) follows from Proposition 5.8 applied to 

1 ^ Z2 ^ fin(5^) ^ Modsi 1, 

where S"^ is the sphere with n boundary components (see [97]). 

Under the supposition of f.g. the most part of groups from (6) is covered by Theorem B 
being residually finite (by Mal'cev's theorem) and containing F2 C SL{2, Z) for characteristic 
0. For a finite characteristic one can find an embedded free product * Z„ (in the cases 
covered by [15, Theorem 10.3]). For characteristic ^ 3 one has m — 1^2, n — 1^2, and 
Zm * Z„ is a Powers group, in particular, non-amenable (see [19, Corollary 12]). 

The new classes, related to this item are pure braid groups (as it is explained in Intro- 
duction) and (full) mapping class groups for compact non-orientable surfaces (with some 
low-genus exclusions). Indeed, let iV be a compact non-orientable surface, such that its ori- 
enting cover S is neither a sphere with ^ 4 boundary circles, nor a torus with ^ 2 boundary 
circles, nor a closed surface of genus 2. Then Mods is C*-simple [13] and residually finite. 
Hence, Modjsi is non-amenable [19, Prop. 3] and residually finite being a subgroup of Mods 
(see, e.g. [3, Sect. 2.1]). 

Another new class is braid groups -Bn(S) and pure braid groups -P„(S) of a compact 
connected oriented surface S (for n ^ 2). Remind, that Pn(S) is defined as the fundamental 
group of -F„(S) := {(xi, . . . ,x„) G | 7^ Xj ii i ^ j}. The symmetric group Sn acts on 
F„(S) by permutation of coordinates, and the fundamental group of the orbit space is the 
braid group i?„(S). We obtain a regular n\ fold covering and the exact sequence 

1 ^ P„(S) ^ fi„(S) ^Sn^l. 

Thus, Theorem B is applicable to i?n(S) if and only if it is applicable to Pr!,(S). We have 
([9], see also [91]) an epimorphism Pm(S) —)■ 7ri(S)™'. If S has genus ^ 1 and a border, then 
7ri(S) is free non-abelian (homotopicaly 1-dimensional complex with at least 2 loops). Thus 
P„(E) is non-amenable in this case. For a closed surface (still different from S"^ and MP^) 
one can consider the short exact sequence from [26] 

1 ^ 7ri(S \ {xi, . . . , x„) ^ P„(S) ^ P„_i(S) ^ 1 

to prove non-amenability for n ^ 3 (now a free group is a subgroup). Non-amenability, 
as well as the residual finiteness in the case of a closed surface can be obtained from [51, 
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Theorem 5] (the residual finiteness can be also obtained by [4]). The residual finiteness for 
surfaces with boundary follows inductively from [51, Theorem 6] and [80, Theorem 7(b)]. 
The several remaining cases should be treated individually. 

Artin's groups Ar (for a graph F) are naturally included in the exact sequence 

PAr ^ Ar ^ Wr ^ 0, 

where PA^ is the corresponding pure Artin's group and Wr is the corresponding Coxeter 
group. Evidently, Ar is non-amenable (containing the free group). If it is residually finite 
(e.g. if it is of finite (or spherical) type (i.e. I^rl < oo), or Wr is free abehan, see [80, 
Theorem 7(c)]), we can apply Theorem B. Also, Ar has the Roc property, if PAr is a 
characteristic subgroup and Wr is neither finite nor almost abelian, because (as we have 
proved above) in this case Wr has the Roo property. Actually, it is proved that PAr is 
characteristic for the finite type case [41]. To compare this with [67] is a separate problem. 

8.3. Known examples of R((j)) < oo, Reidemeister spectrum, and Theorem A. Now 
we would like to list some cases, where Theorem A is applicable. 
Following [95] , define the Reidemeister spectrum of G as 

Spec{G) := {A; G N U cx) I R{(p) = k for some E Aut(G)}. 

In particular, _Roo Spec{G) = {oo}. 

It is easy to see that Spec{Z) = {2} U {oo}, and, for n > 2, the spectrum is full, i.e. 
5'pec(Z") = NU{oo}. For free nilpotent groups we have the following: Spec{N22) = 2NU{oo} 
{N22 is the discrete Heisenberg group) [63, 30, 95], Spec{N2-i) = {2k'^\k G N} U {00} [95] and 
Spec{N32) = {2n - l\n G N} U {4n\n G N} U {00} [95]. 

Metabelian non-polycyclic groups have quite interesting Reidemeister spectrum [38]: for 

fr 0\ 

example, if ^(1) is of the form ( ^ ^ 1 , then we have the following cases: 

a) If r = s = ±1 then S'pec(Z[l/p]^ Z) = {2n\n G N, {n,p) = 1} U {00} where {n,p) 
denote the greatest common divisor of n and p. 

b) If r = -s = ±1 then Spec{Z[l/p]'^ Z) = {2p'(/ ± 1), 4p'|/, A; > 0} U {00}. 

c) If rs = 1 and |r| ^ 1 then Spec{Z[l/p]'^ xig Z) = {2(p' ± 1),4|/ > 0} U {00}. 

d) If either r or s does not have module equal to one, and rs ^ 1 then Spec{Z[l /p]^ xigZ) = 
{00}. 

An interesting case was studied in [34], where the Reidemeister number and the number 
of fixed points of where compared directly. In this example G is a semidirect product of 
Z^ and Z by Anosov automorphism defined by the matrix ^ ^ | ^ G is solvable and of 

exponential growth. The automorphism is defined hj ^ ^ on Z^ and as — Id on Z. 

Now we will show examples of some groups with extreme properties, where -R(0) < 00 
but Theorem A is not only unapplicable, but the statement is wrong. Namely, Osin's group 
[90] has exactly 2 ordinary conjugacy classes, thus it is not residually finite. Hence, by 
Mal'cev's theorem it is not linear. Thus, it has no finite-dimensional representations with- 
out kernel. Since it is simple, there is no representations with non-trivial kernel (only the 
trivial 1-dimensional representation). Thus, R(ld) = 2, but the (fixed) finite-dimensional 
representation is unique. A similar argument works for Ivanov's group [87, Theorem 41.2] 
and some HNN-extensions [62, 100] (see [39] for details). 
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